A hierarchical extension to 3D non-parametric surface relief completion

Typical stereo and laser scan based 3D acquisition approaches are essentially limited to 2.5D capture. The resulting 3D completion problem, to derive missing information in 2.5D scenes from limited contextual information, has received increasing attention in literature. Here we propose a hierarchical extension to our recent nonparametric approach for the 3D completion of surface relief detail to allow the resolution of inconsistencies arising in the global structure of an area completed with this technique. We test our approach over a range of surface types and contrast the presence of global discontinuities in the resulting completion with those of the earlier approach. 1 3D Surface Completion Over recent years the problem of visual completion has become of increasing interest in both 2D and 3D imaging. A large proportion of work in this area can be set in the context of human abilities within the domain of perceptual psychology [6]. Specific to the completion problem faced in 3D computer vision is the 2.5D limitation of current 3D sensing approaches. With a single laser scan or stereo capture it is not possible to capture all faces of a 3D scene. Essentially, from this uni-directional viewpoint we have a 2.5D scene capture 3D information but in one single direction. The result is 3D surface models that are inherently 2.5D in nature or 3D models which require construction from a combination of multiple uni-directional captures. In this latter case, additional capture and viewpoint combination incur significant additional work [3]. Following on from our work in [7] we propose a hierarchical approach that builds up the missing information within unknown scene portions as an iterative process from global structure to localised surface detail. This in turn follows the paradigm of visual propagation as the basis for the completion of 3D scenes via the propagation of knowledge from known to unknown scene Preprint submitted to Elsevier February 28, 2011 portions [6]. Specifically we propose a multi-level hierarchical approach for the nonparametric synthesis of 3D surface relief from an original sample surface region to a co-joined target region. This approach uses a 3D extension of nonparametric texture synthesis [13] in conjunction with knowledge derived about the underlying shape continuity of the surface [28,8,11]. Whilst the latter can be achieved via a variety of approaches [28,8,11,10,21,33,17,31] we rely on a simple geometric fitting approach to provide the required surface knowledge to illustrate our technique. As shown by our results, the resulting approach overcomes the limitations in global surface structure continuity (e.g. Figure 3) found in earlier work [7]. Prior work in this area can be divided into two areas completion via good continuation, and completion via structure propagation. Work within the area of good continuation concentrates on the smooth completion of underlying surface. Several approaches have been proposed within this context including localised geometric and algebraic surface fitting [28,8,11,9,18], volumetric distance-field based techniques [10,32,23,15], cross-triangulation patching and refinement [21,33,31] and spatial occupancy based approaches [24,17]. Whilst these techniques concentrate on the completion of the global underlying surface shape more limited attention has been paid to the additional topic of propagating surface detail thus limiting them in application with real-world surfaces [6]. By contrast work in the area of structure propagation concentrates on the concept of “completion by example”. Structural propagation techniques, by the nature of the problem, frequently draw upon the success of 2D work in the analogue problem of 2D texture synthesis. The approach proposed here is no different. Notably the prior work of [27] pursues a patch-based “copy and paste” completion approach akin to [12] that, whilst well suited to smooth surface or irregular/anisotropic relief completion, relies on non-rigid alignment (warping) that limits its application to structured relief (as found in architecture, e.g. Fig. 2D). The brittleness of the patch-wise “copy and paste” approach can also lead to uncharacteristic ‘tiling’ artifacts over large areas and an inability to adaptively complete in scenarios where no suitable propagate patch exists [27]. By contrast the “example-based” completion work of [25] is performed with reference to a database of similar a priori complete 3D objects from which similar surfaces are selected and blended to perform completion on the partial 2 2 D surface. This is clearly limited by suitable a priori knowledge and, by its use of similarity matching, relates more to the concept of completion via recognition than the completion via generalisation we aim for here. The related works of [2] and [19] are similar respectively to [25] and [27] but vary primarily in underlying surface representation, and the graphics work on geometric texturing [4,20] covers a related but not explicitly completionorientated problem domain. Both [4] and [20] are so far limited to arbitrary